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ABSTRACT 

The 1 imitations of objectives tests , the language 
skills needed for university work and effective ways of appraising 
and communicating these skills to the university are discussed. The 
goals of university English as a second language (ESL) programs is to 
train international students in the linguistic and academic skills 
they will need in American universities . This goal is complicated by 
objective language tests Such factors as cheating, test coaching , 
testing emphasis upon language recognition and not production , 
cultural bias, test interpretation, and test anxiety often cause a 
discrepancy between test scores and actual ESL potential. Some of the 
skills that are needed by university students but are inadequately, 
tested fry objective tests. include reading, writing, listening, oral 
production, and study skills. It is necessary to evaluate the 
student's proficiency in these skills in addition to an objective 
test score in order to determine University admittance . Therefore , 
ESL teachers should insist that testmakers continue to improve the 
objective tests and to promote the acceptance of skill-based 
evaluations of students 1 proficiency by admission officers. (RW) 



********************************************************* 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document • * 
*********************************************************************** 



ERIC 



» — r 

rvj 
Q 



er 
to 



A Complement to Testing: Ski 11 -Based Appraisal 

Laura Latulippe Mary Lu Light 

The purpose of university ESL programs is to train foreign students to 

perform linguistic tasks in the university. Objective tests used as deter- 
miners of admission to most universities ^ however complicate* and at times 
preclude efforts to fulfill this purpose. The student's goal is often to 
obtain a sufficient score on the MTELP or TOEFL, not to absorb the needed 
ski lis. 

At present, objective tests are used almost exclusively to determine 
university preparedness. Testing experts admit the statistical accuracy of 
these tests is limited, and we question the value of the information they do 
test in determining university success. 

We would like to show the limitations of objective tests, summarize the 
language skills needed for university work, and discuss effective ways of 
appraising and communicating these skills to the university 

We propose that: 

1. Test makers expand their tests to include relevant 
university language skills evaluation, eg. note- 
taking lecture listening. 

2. Language institutes devise a consistent method of 
evaluating and communicating these skills to 
admission people as a complement to improve objective 
tests . 

3. ESL teachers undertake a campaign to make admission 
people aware of objective test limitations and the 
need for using teacher-based evaluations as 
complements . 
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A Complement to Testing: Skill-Based Appraisal 

The goal of university ESL programs is to train international students in 
the linguistic and academic skills they will need in American universities 
This goal is complicated by,, and at times, precluded by the objective language 
tests used to determine international students' admission to universities. 
Because of the emphasis placed on objective test scores * su^h as those of the 
Michigan Test of English Language Proficiency (MTELP) and the Test of English 
as a Foreign Language (TOEFL), a second goal is being imposed upon students 
and university ESL programs. This Ts the goal of obtaining high test scores 
which do not necessarily reflect an ability to succeed in the university. 

The difference between what is actually being tested by these objective 
tests and what is needed in the university, suggests that a high test score 
will not assure success in the university. At best these tests are a limited 
indicator of the students 1 knowledge of English; they do not indicate much 
about the students 1 academic skills. 

We feel that the conflict between the goal of training students for the 
university and this imposed goal of training students to obtain high test scores 
is interfering with our work of helping students to succeed in the university. 
The purpose of this paper is to A) discuss the limits of the tests, B) discuss 
the skills necessary for university work* C) propose an alternative to the 
current situation, and D) outline some suggestions for ESL teachers who wish to 
change the situation. 



ERLC 



LIMITS OF TESTING 

The two most frequently used tests, MTELP and TOEFL contain several com- 
ponents or sub-tests that are averaged together to give a single test score used 
by admissions officers. The components of the currently-used MTELP include; a 
sentence level aural comprehension test, a recognition test of sentence level 
grammar, a recognition test of non-contextual ized vocabulary from the 4000-6000 
word level and a reading comprehension test of paragraphs on subjects often 
unrelated to the academic needs of international students. In addition, the 
official form of the MTELP includes a writing sample and an' oral interview. 

Recently the University of Michigan English Language Institute has revised 
the MTELP and renamed it The Michigan English Language Assessment Battery 
(MELAB) . The MELAB is a step forward in testing English for academic purposes. 
While the composition section remains unchanged, the listening portion now 
contains three parts: aural grammar, aural emphasis, and dialogue/lecture. The 
dialogue/lecture section allows students te take notes on a short lecture or 
dialogue and use these notes to answer multiple choice questions about a visual 
related to the talk. The third part of the test contains a multiple choice 
grammar section, a multiple choice cloze passage, a vocabulary section and 
reading passages. The format of the reading passages is the same, but the 
content has been changed to include natural science, popular psychology, tech- 
nology subjects and biology. This new content is a more appropriate test of the 
type of reading that foreign students will do in their university classes. The 
MELAB is an improvement over the old MTELP, but it is currently used only at the 
University of Michigan. 

The TOEFL includes a listening comprehension test containing three sub- 
sections: a statement requiring that the examinee choose a statement that is 
similar in meaning, a short conversation followed by a question for which the 
examinee must choose the answer, and short talks followed by questions, for 
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:which the student must choose the answer. It also has a section of structure 
and written expression containing two sub-sections: a test of sentence level 
grammar recognition and a recognition test of non-acceptable words or phrases 
in sentence level written material. The third sub-section is reading compre- 
hension and vocabulary, including sentence level recognition of vocabulary and 
questions about academically oriented paragraphs. 

The objective tests do no4 test oral production; textbook reading; the 
use of the library and other academic resources; the writing of academic 
papers or essay test answers, which are needed for university success; and 
lecture listening, perhaps the most important language skill, is tested only 
in a limited way. 

There are several factors that limit objective tests in their ability to 
determine university preparedness, some of which are inherent in the tests 
themselves, and others which are a result of the way in which the test scores 
are interpreted. These factors fall into two categories: those which unjust- 
ifiably raise scores and consequently help students gain admission when they 
are not ready, and those which lower scores and prevent capable students from 
being admitted to the university. We will consider each category separately. 

One of the factors that raises scores unjustifiably is bad test security. 
This includes both cheating during the exam and obtaining a copy of the test 
before it is given. Although the major test producers have made progress in 
eliminating most of these problems, isolated cases do exist. 

Scores can also be raised by practice and coaching. Students who take 
the test frequently, will usually increase their scores even if their know- 
ledge of English has hot increased. In addition, coaching for these tests can 
raise the score. After taking a special TOEFL preparation class (200 hours) at 
Florida State University, students were able to rarse their scores an average 
of 106 points. (Jenks, 1980) Even when the test score is not significantly 



raised by this coaching, the problem is that students put time and energy into 
practice and coaching believing this will help, time and energy that should be 
spent learning the academic English skills they will need after they are admitted. 

A third score-raising factor, possibly the most important in terms of 
university admission, is that objective language tests can measure only the 
recognition of language and not the production of it. We, as language teachers 
are aware of the difference, but those who interpret the scores and use them to 
determine admittance are not always aware. 

The final score-raising factor is the guessing effect, which, although it 
is not a major problem, should be considered when test scores are Used to . ' 
indicate a student's readiness for the university. 

Cultural bias is the first factor that unjustifiably lowers scores and 
prevents deserving students from entering the university. This includes bias 
in the test material and in the format of the test. The reading comprehension 
section of objective language tests frequently contains paragraphs dealing with 
American history and literature. Few international students plan to study in 
these areas. The Institute of International Education's 1980-81 Census of 
Foreign Students indicates that the largest percentage, 25.8% of international 
students in the U.S. are studying engineering. The second most popular subject 
is business and management accounting for 17.4% of the students, followed by 
8.1% who are undecided about their area of study and 7.8% studying social 
science. (Boy an, 1981) Why should they be asked to demonstrate competence in 
reading subject matter to which they may have only marginal exposure in their 
university work? Another aspect of cultural bias is the format of the tests. 
Students who come from countries that do not use objective testing methods are 
at a disadvantage when taking this type of test for the first time. 

The second factor that lowers scores and hinders admission is score inter- 
pretation. Due to the statistical error of measure (SEM), an objective test 
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score does riot represent an exact rcore, but rather a range within which the 
score falls; The TOEFL, for example, has a SEM of 16 points. Admissions 
officers who do not know this often set a specific score cut-off for admission, 
and, as a result they don't admit students who have scores falling in the same 
range as students who are admitted. 

The last factor contributing to lower scores is test anxiety. Because 
these test scores are frequently the final determiners of admission, and there- 

i ... 

fore of students' futures, they experience much anxiety during testing. Studies 
indicate that high anxiety can lower test scores. (Madsen, 1981) The tests 
that are used to determine our students' futures, are limited in their ability 
to measure students' English and, do not test the skills Which may be the most 
necessary for success in the university. Language production is difficult to 
test reliably, but this does not mean that the productive skills can be ignored. 
They must be considered to determine students' ability to succeed in American 
universities. 

SKILLS NEEDED FOR THE UNIVERSITY 

During the process of revising the curricula for our various programs , 
many ESL teachers have completed heeds assessment to determine what language 
skills are necessary for the study areas of their students. The results of 
several of these assessments have been published* (Ostler, 1980} (Lee, 1979) 
and our own survey had similar results. Many skills that are heeded by our 
students in the university are inadequately tested or not tested at all by the 
current objective tests. These skills can be divided into five categories; 
reading skills* writing skills, listening skills, oral production skills and 
study skills. 

Rather than reading isolated paragraphs, as students must prove their 
ability to do on objective tests, our students in the university must have all 
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of the skills necessary to read college level textbooks arid scholarly articles. 
They must be able to Identify main ideas and relate them to supporting details, 
to skim, scan, understand reference words and connectives, make inferences, 
distinguish fact from opinion, use context clues and word analysis to guess the 
meanings of new Words, recognize the function of a text and the point of view 
arid plan of authors, synthesize the text with previous knowledge and synthesize 
information from various sources. (Hamp-Lyons , 1982) 

In the university, our students need to do more than write unified, 
coherent pa ragraphs. They .nust be able to wri te rapidly, pa raphrase ideas that 
they hear in lectures as notes , and use these notes to write clear, essay 
question answers within the time limit of a fifty minute class. Most of our 
students will be required to write summaries, abstracts , and term papers all of 
which require skill in pa raphrasing and organizing ideas. Finally, some students 
are required to produce lab reports, research proposals, critiques and resumes. 
Each specialized form of writing could, of course, be learned after admittance 
to the university, but, without the complex writing skills that underlie all of 
these forms of writing, students have a disadvantage competing with their 
American counterparts. 

The most important group of listening skills are those needed during class 
lectures. The student needs to recognize the organization of lectures; listen 
for the important ideas (generalizations) , which entails understanding 
rhetorical phrases (consequently, the former, on one hand), and recognizing 
verbal and non-verbal cues to key ideas (a change in voice, a gesture, or a 
change in movement); and adjust to digressions by the speaker, unfamiliar idioms , 
and poor spea kers. In addition to formal lecture material , students must be 
able to understand class discussions , as important questions are often asked and 
answered during the exchange of ideas in a class. 
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Oral production is also necessary for success. Students must at least be 
able to ask questions of instructors about lectures and reading material; and 
many students find it necessary to participate in class discussions from which 
they are p.-rtial ly evaluated. Many, especially graduate students, must be able 
to give well -organized and clearly stated oral reports in class, which is 
different from merely reading a paper. International students need to be 
especially obvious in their organization and clear in pointing out key ideas, 
as they have the added burden of an accent to which the instructor and other 
students may have trouble adjusting. 

The final category of skills consists of language-related study skills 
needed by all students in any study area. These include using the library 
effectively; using a dictionary and thesaurus to check spelling, meaning, and 
exact usage; and being able to proof-read arid correct one's own papers. Our 
students also need to know how to take several varieties of tests (essay, 
multiple choice, fill-in) , and they must be able to use language effectively 
(including body language and levels of language) to get help from fellow 
students* instructors, arid librarians. 

EVALUATING SKILLS 

As we have determined the needs of our students* we ESL teachers have 
created materials to teach the needed skills. We have written books that teach 
students to write for academic purposes, arid We have written many excellent 
reading books that teach our students all of the skills needed for academic 
reading. As we use these books and add to them our own exercises arid tests, 
we evaluate the students' progress. Our evaluation of their skills Consists 
of various types of assessment over a long period of time rather than a two 
hour objective test, the outcome of which i.- influenced by even more variables 
than those operating in the classroom. 
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The skills that are needed to write a term paper, for example* can best be 
evaluated during the proeess of writing it so that each step can be evaluated. 
The teaeher learns if students have adequate library skills by interacting with 
them as they use the library. Even the one skill of paraphrasing needs to be 
both taught and evaluated step-by-step, from recognizing a paraphrase in an 
exercise, to using connectors to paraphrase sentences * to the final step of 
paraphrasing whole sequences of ideas arid synthesizing ideas from several 
sources. The process of editing the paper is aided arid evaluated by the teacher 
working with the student, and the final paper can then be graded and compared 
with university standards. This total evaluation determines a student's ability 
to produce a term paper- as accurately as possible. 

Oral reports can be evaluated while they are presented in class, with forms 
that focus oh all the points of a good oral report- clear purpose, well- 
substantiated points, good use of connectors, etc. One such form was suggested 
by Christine Meloni and Shirley Thompson. (Meloni arid Thompson, 1980} In 
addition to teacher evaluation, the audience of students can be asked to show 
their understanding of the points presented by filling in a very basic outline 
of the report. If members of the audience cannot determine the main points, 
the spea ker knows that the points were not clearly made. 

ESP classes can be excellent places to evaluate several skills. Textbook 
reading in a specific study area can be taught and evaluated by using actual 
materials , teaching and evaluating knowledge uf rhetorical style* vocabulary, 
text organization, and the use of the bibliography and footnotes in addition 
to general comprehension. The material can be discussed in class allowing the 
teacher to evaluate students' ability in class discussion and questioning. And, 
finally, ability in different types of test-taking can be determined by testing 
students 1 knowledge of subjects studied using each form of test. 
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COMMUNICATING EVALUATIONS 

We feel that an evaluation of the students' proficiency in the above- 
mentioned skills should be used in addition to an objective test score to 
determine university admittance. One way of communicating this evaluation is 
a skills checklist. This list could contain the skills heeded in the university 
and an evaluation by the students' teachers of their level of attainment in each 
skill. Admission officers could decide what level is needed in each skill area 
for admittance, depending on the student's area of study. The checklist at the 
end of this paper has been used to evaluate students at our Advanced level. 
This form contains the student's objective test scores, but it emphasizes teacher 
evaluation of the skills necessary to the students' academic future. The skills 
checklist Can be revised as the program curriculum is revised. 

There. are, of course, problems that can arise with using something as 
"subjective" as a skills checklist, but we think they can be satisfactorily 
dealt with. The decision of whether or not a student can perform a certain 
skill may vary from teacher to teacher, depending on the teacher's understanding 
of what is meant by this skill and how accomplished a student must be at the 
skill in order to succeed in the university. But the checklists could be 
standardized and a clear description of what is meant by each level and each 
skill could be provided. Actual samples of satisfactory work in addition to 
written descriptions should make the standard clear. Al so teachers who will be 
doing the evaluation could be required to attend workshops that would train them 
to use the Checklists in a standard way. 

Another problem is that admission people might interpret the skills check- 
list differently. But again, they could be trained and assisted as the teachers 
are to make meaning clear. A final suggestion to avoid possible problems is 
that the evaluation of university skills should be done at the advanced level 
of a program, using university level materials so that the evaluation has 



meaning and can better be used to predict future success. The fact that a 
student is able to comprehend the main ideas and relate them to supporting 
ideas in reading material at an intermediate level, does hot necessarily mean 
that the student can do the same thing with a college text chapter. 

WHAT YOU CAN DO 

As ESt teachers there are several things we can do to insure that our 
future students will be evaluated more completely and fairly, and to help our 
own programs by clarifying program goals. First, we can insist that testmakers 
continue to improve the objective tests. Admissions officers will probably 
continue to rely heavily on objective tests in the future, because they deal 
with Targe numbers of people and heed a fast and efficient means of determining 
university preparedness. Therefore, it is vital that the major test producers 
aim at improving the tests to include relevant skills. 

Future tests should be designed to test English for academic purposes and 
not only general proficiency in English. These tests should include subject 
matter from all the major academic areas: such as the humanities , sciences , 
and business. To whatever extent it is possible whi 1 e retaining validity, 
rea liability, and practicality, they should test language skills and not merely 
language items , particularly in the area of vocabulary. They should include a 
sample of the student's written language while performing an academic task, which 
could be forwarded to the admitting university along with the other test scores. 

An acceptable objective test is a goal that will not be achieved in a short 
time, but it is imperative that testmakers pursue this goal. 

The second step ESL teachers can take to improve the current process is 
to work toward getting ski 11 -based evaluations of students' proficiency 
accepted by admission officers in addition to objective test scores, fo do this, 
we need to work together to create standardized checklists for intensive English 
programs to complete and send to admission offices along with test scores. This 



ta * k deludes not o nly making the list, but also writing the descriptions of each 
5kr M arid level and agreeing upon student examples that fit each category 
fondly, we need to se t up workshops to train teachers to use the checklists 
aMCl us e them in 0ur programs to test and revise them. Then we need to work with 
the university admission people to get them to accept these checklists. We need 
t° Mnt out to them the inadequacies of objective tests as the scle predicters 
° f diversity suce e $ s . We need to show them the standardized checklists complete 
w1th descriptions a n d examples, explain that teachers have been trained to use 
tm effectively* arid offer to let them observe classes in which this on-going 
equation process t s occurring. This personal lobbying approach can help to 
be* 1 Id the trjst that i s necessary to allow skill-based evaluation to work. 

University £SL programs should be training international students in the 
H' n 9tnstic and acad^ic skills they need to succeed in American universities. 
Bemuse attitudes to War d current objective tests hinder this goal in addition 
to fusing difficult problems for our students, we would like to change the 
amission policies b eing use d today. We make the preceeding observations and 
suggestions with the hope that we can all work toward this end. 
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LEVEL: | APVAMCEOtf-J STUDENT'S NAME; 



THE CAREER ENGLISH LANGUAGE CENTER 

F 0 * INTERNATIONAL STUDENTS - W M U PROGRESS RETORT 


_ RESEARCH CLASS Instructor: y^R HARE 


ACCEPTABLE FOR COLLEGE WORK: 


Attendance: hours present: hours absent: 


NO 


ALMOST 


YES 


Able to: -adequately paraphrase materials for research: 








- w r1te a clear, logical outline of material gathered 

for research: 

-locate various research sources In the library: 








-write a we 11 -organ 1 zed ana documented 4 page research 
paper: 








-give an oral report^based-of^fcfm research paper: 
Comments: FCLASS EVALUATION FqR ADVANCED LEVEL: 









WRITING CLASS INSTRUCTOR: DLOUHY 


ACCEPTABLE FOR COLLEGE WORK: 


Attendance: Hours present: Hours absent: 


NO 


ALMOST 


YES 


Able to: -paraphrase col leqe level readlnq material 1n wrltlnq: 








-summarize colleqe level reading material in writing* 








-write a clear, well-organized paper with an 
1ritroduct4on. well -supported body and conclusion: 








-usejEngMsh grammar and sentence structare 
adequately: 

-edit and correct own work: 

COMMENTS: iLLAkk kVAHiAliliW H1H AlWAWtikU Lll/EL: 









READING CLASS 
Attendance: 



INSTRUCTOR: 



KEESLER 



Hours present: 



Hours absent: 



flfefe to: 



-read and compr ehend college leVfrl^mat eH^H 
-read college Te^e^ma^wST^I tTf sufficient" st 




ACCEPTABLE FOR COLLEGE WORK 



NO 



ALMOST 



YES 



SPEAKING AND LISTENING CLASS INSTRUCTOR: LATUL IPPE 


ACCEPTABLE FOR COLLEGE WORK * 


Attendance: Hours present: Hours Absent: 
Able to: -understand the main ideas of a short lecture- 


NO 


ALMOST 


YES 


-organize lecture notes and use them effectively 

i ri-a-^«t-s+tu*t 1 on : 
rex press Ideas clearly in qroop discussions^- " 








Comments: ICLASS EVALUATION FOR Anvtti^h t fvfi ■ 









PROFILE: Academic motivation and willingness to 1 



LEVEL COORDINATOR: 



LATULIPPE 
MICHIGAN TEST RESULTS: 



rtecp ism 




Not valid without seal. 



